Visual content overlay system

ABSTRACT

An augmented reality display system included in a vehicle generates an augmented reality display, on one or more transparent surfaces of the vehicle. The augmented reality display can include an indicator of the vehicle speed which is spatially positioned according to the speed of the vehicle relative to the local speed limit. The augmented reality display can include display elements which conform to environmental objects and can obscure and replace content displayed on the objects. The augmented reality display can include display elements which indicate a position of environmental objects which are obscured from direct perception through the transparent surface. The augmented reality display can include display elements which simulate one or more particular environmental objects in the environment, based on monitoring manual driving performance of the vehicle by a driver. The augmented reality display can include display elements which identify environmental objects and particular zones in the environment.

This application is a divisional of U.S. patent application Ser. No.15/762,543, filed Mar. 22, 2018, which is a 371 of PCT Application No.PCT/US2016/053182, filed Sep. 22, 2016, which claims benefit of priorityto U.S. Provisional Patent Application No. 62/232,855, filed Sep. 25,2015. The above applications are incorporated herein by reference. Tothe extent that any material in the incorporated application conflictswith material expressly set forth herein, the material expressly setforth herein controls.

BACKGROUND

In many situations, a graphical overlay can be provided on anenvironment that is perceived through a transparent surface, including awindow. A graphical overlay can provide information to an observer,including information which is relevant to one or more portions of theperceived environment. In some cases, a graphical overlay is used in avehicle, where a graphical overlay can be perceived by an occupant ofthe vehicle and provides information relevant to one or more of thevehicle, including vehicle speed, and one or more portions of theenvironment in which the vehicle is located. Such information can beprovided on a transparent surface, including a forward window, alsoreferred to herein as a windscreen, windshield, etc., such that theinformation can be perceived by an operator of the vehicle, including adriver, near the line of sight of the operator as the operator observesthe environment through which the vehicle is navigated, including anoncoming roadway.

In some cases, the graphics displayed should be able to overlay certainobjects located in the environment perceived though the transparentsurface and can augment, alter, control, information related to theenvironment which is provided to an observer via the transparentsurface. Such graphical overlays can be referred to as an augmentedreality display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a vehicle, according to some embodiments.

FIG. 2A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display indicating a spatial indication ofa speed of the vehicle relative to a local speed limit, according tosome embodiments.

FIG. 3A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which modifies occupant-perceivedinformation presented by a sign located in the external environment,according to some embodiments.

FIG. 4A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which provides a representation ofa static object in the environment which is presently obscured fromdirect view through the transparent surface of the vehicle, according tosome embodiments.

FIG. 5A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which provides a representation ofat least one dynamic object in the environment which is presentlyobscured from direct view through the transparent surface of thevehicle, according to some embodiments.

FIG. 6A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which provides a dynamicallycontrollable representation of a driving route along which a vehicle isbeing navigated, according to some embodiments.

FIG. 7A-B illustrate a vehicle which simulates, via one or more of atransparent surface which provides an augmented reality display and anactive suspension system, one or more objects in the environment,according to some embodiments.

FIG. 8A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which indicates a particulardriving zone in the environment, according to some embodiments.

FIG. 9 illustrates a transparent surface which provides an augmentedreality display of one or more portions of an environment, according tosome embodiments.

FIG. 10 illustrates a vehicle comprising a transparent surface whichprovides an augmented reality display which enables user selection of aportion of the environment perceived via the transparent surface andwhich enables video communication with a remotely-located user via thetransparent surface, according to some embodiments.

FIG. 11 illustrates an example computer system that may be configured toinclude or execute any or all of the embodiments described above.

DETAILED DESCRIPTION

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the appended claims,this term does not foreclose additional structure or steps. Consider aclaim that recites: “An apparatus comprising one or more processor units. . . .” Such a claim does not foreclose the apparatus from includingadditional components (e.g., a network interface unit, graphicscircuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware—for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112, sixth paragraph, for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software and/or firmware(e.g., an FPGA or a general-purpose processor executing software) tooperate in manner that is capable of performing the task(s) at issue.“Configure to” may also include adapting a manufacturing process (e.g.,a semiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On.” As used herein, this term is used to describe one or morefactors that affect a determination. This term does not forecloseadditional factors that may affect a determination. That is, adetermination may be solely based on those factors or based, at least inpart, on those factors. Consider the phrase “determine A based on B.”While in this case, B is a factor that affects the determination of A,such a phrase does not foreclose the determination of A from also beingbased on C. In other instances, A may be determined based solely on B.

FIG. 1 illustrates a vehicle, according to some embodiments. The vehicle110 is located in an external environment 100 and can include a vehiclenavigation system (VNS) 140. In some embodiments, the VNS 140 canautonomously control one or more vehicle control elements 150 to causethe vehicle 110 to be autonomously navigated along one or more drivingroutes through the environment 100. In some embodiments, VNS 140controls display of information to one or more occupants of the vehiclevia one or more interfaces 130. At least one of the interfaces 130 caninclude a transparent surface via which an occupant can perceive one ormore portions of the environment 100, and the VNS 140 can control adisplay of information to the occupant via a graphical overlay on thetransparent surface which provides an augmented reality display of oneor more portions of the environment perceived by the occupant via thetransparent surface.

Vehicle 110 includes one or more sets of external sensors 116 whichgenerate sensor data representations of one or more portions of theexternal environment 100. The external sensors 116 can include one ormore of visible light camera devices, infrared camera devices,near-infrared camera devices, light beam scanning devices, ultrasonicsensor devices, audio sensor devices, depth camera devices, radardevices, geographic position detection devices, some combinationthereof, etc. Sensor data representations generated by one or more ofthe external sensors 116 can be communicated to the VNS 140, and the VNScan generate and maintain a world model of one or more portions of theenvironment 100 based on the sensor data representations generated bythe external sensors 116. In some embodiments, the VNS 140 generates andmaintains a world model based at least in part upon information receivedfrom a remote service, system, device, etc. via one or morecommunication networks.

Vehicle 110 includes a set of one or more internal sensors 118 whichgenerate sensor data representations of one or more portions of avehicle interior included in the vehicle 110, including sensor datarepresentations of one or more occupants of the vehicle, also referredto herein as users. The internal sensors 118 can include one or more ofvisible light camera devices, infrared camera devices, near-infraredcamera devices, light beam scanning devices, ultrasonic sensor devices,audio sensor devices, depth camera devices, radar devices, somecombination thereof, kinematic sensor devices, etc. In some embodiments,one or more internal sensors 118 generates sensor data associated with astate of one or more portions of the vehicle, including one or more ofthe control elements 150. Such one or more internal sensors can includeone or more of an accelerometer, a speedometer, etc.

Vehicle 110 includes one or more control elements 150 which cause thevehicle to be navigated. Control elements can include one or more of asteering control element, a motor, a throttle control element, a brakecontrol element, etc.

Vehicle 110 includes one or more interfaces 130, which can include oneor more user interfaces installed in the vehicle with which one or moreoccupants can interact, one or more communication interfaces which cancommunicate with one or more remotely located devices, vehicles,services, systems, etc. via one or more communication networks. Vehicle110 includes one or more transparent surfaces 134 via which one or moreoccupants can perceive one or more portions of the environment 100 fromwithin the vehicle interior, etc. In some embodiments, VNS 140 cangenerate, manage, control, etc. one or more graphical displays,including one or more augmented reality displays, which are displayed onone or more particular transparent surfaces 134 in the vehicle.

As shown, the VNS 140 can include an augmented reality display module111, also referred to herein as an augmented reality display system. Insome embodiments, one or more of the VNS 140 and the module 111 can beimplemented by one or more computer systems. In some embodiments, theVNS 140 and the module 111 are implemented by separate computer systems.Module 111 can generate one or more various augmented reality displayswhich are provided to one or more occupants via one or more transparentsurfaces 134 in the vehicle 110 based on information received viaprocessing data received from one or more of the interfaces 130,external sensors 116, and internal sensors 118.

FIG. 2A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which includes a spatiallypositioned representation of a speed of the vehicle relative to a localspeed limit, according to some embodiments. The vehicle and transparentsurface illustrated in FIG. 2A-B can be included in any of theembodiments of vehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display provided to anoccupant of a vehicle, via being displayed on one or more transparentsurfaces of the vehicle, includes a display element which provides athree-dimensional representation of a speed of the vehicle whererepresentation of the speed is spatially positioned, in the environment,based on a difference between the vehicle speed and a local speed limit.The vehicle can include an augmented reality system which, based onsensor data generated by one or more internal sensors, external sensorsinterfaces, etc, can determine a present speed of the vehicle and alocal speed limit in the portion of the environment through which thevehicle is presently being navigated.

The augmented reality display system can generate an augmented realitydisplay on a transparent surface of the vehicle which includes a displayelement which both indicates a speed of the vehicle and provides atleast a spatial indication of the speed of the vehicle, relative to thelocal speed limit. In some embodiments, the display element includes arepresentation of at least the speed of the vehicle. The augmentedreality display can comprise an overlay of at least a portion of theenvironment visible, also referred to herein as perceptible, via thetransparent surface. The overlay can include display elements, alsoreferred to herein as representations, which provide three-dimensionalrepresentations of one or more various graphical icons, such that thethree-dimensional representations are perceptible as being positioned inthe external environment when the environment is perceived via theaugmented reality display presented on the transparent surface. Adisplay element which provides a three-dimension representation of agraphical icon at a particular position in the environment can bereferred to as being spatially positioned in the environment and can bereferred to as being spatially positioned. Where the display elementprovides a three-dimensional representation of a graphical icon in aparticular position in the environment based on relative vehicle speed,the display element can be referred to as providing a spatial indicationof the relative vehicle speed. Spatial positioning of a representationpresented by a display element in the augmented reality display can beimplemented via controlling one or more of the size and position of thegraphical icon comprised in the display element presented on thetransparent surface, so that the graphical icon can appear to be furtheraway from the transparent surface, nearer to the transparent surface, indifferent portions of the environment, etc., based on the relative speedof the vehicle.

As shown in FIG. 2A-B, a vehicle 200 in which an occupant 210 is locatedcan include a transparent surface 220, including a windshield, via whichthe occupant perceives at least a portion of the external environment290. The occupant 210 can be a driver of the vehicle 200, positioned ina driver position in the vehicle, some combination thereof, etc., andthe transparent surface 220 can include a windshield via which theoccupant can perceive a portion of the environment 290 which is ahead ofthe vehicle along a direction of travel while the vehicle navigatesalong a roadway 292 in the environment.

As shown, an augmented reality display 230 can be displayed(“presented”, “provided”, etc.) on surface 220, so that the display 230is perceptible by occupant 210 monitoring the portion of the environment290 via the surface 220 and the display 230 is perceptible as an overlayupon the environment perceptible by occupant 210 via surface 220. Asshown in FIG. 2A-B, the display 230 can include one or more displayelements 232A-C which include content which indicates at least a speedof the vehicle, such that the display elements comprise representationsof at least the speed of the vehicle. For example, element 232Aindicates a speed of 40 mph, element 232B indicates a speed of 35 mph,and element 232C indicates a speed of 30 mph. In some embodiments, theillustrated elements 232A-C indicate different states and configurationof a single element when the vehicle 200 is navigating at differentspeeds, such that the individual element is perceptible as beingspatially positioned at an individual position in the environment.

As further shown, in some embodiments, the element 232A-C can includecontent indicating a local speed limit associated with the environment,including the local roadway 292. FIG. 2A-B shows that the local speedlimit is 35 mph, and the display 230 includes an element 232A-C whichindicates both the present speed of the vehicle and the local speedlimit.

In some embodiments, the one or more display elements 232 presented viadisplay 230 are adjustably controlled to cause the perceived elements232 to provide three-dimensional representations of one or moregraphical icons which are spatially positioned in one or more portionsof the environment perceptible through surface 220 based on the relativespeed of the vehicle relative to a local speed limit. The spatialpositioning can include adjusting one or more various parameters of theone or more display elements 232, including one or more of a size,color, position, etc. of the elements 232 in the display 230, to causethe element 232 to be perceived as being spatially positioned in theenvironment 290 based on the relative speed of the vehicle 200. Theparameters of an individual display element 232 can be dynamicallyadjusted based on variations in the vehicle speed, local speed limit,etc., so that the three-dimensional representation of a spatial positionof the graphical icon in the environment is dynamically adjusted.

As shown in FIG. 2A-B, the spatial positioned representation of an iconin the environment by the display element 232 in the display 230, basedon the relative speed of the vehicle 210, can include adjustablycontrolling one or more parameters of the display element to cause therepresentation of the icon to be spatially positioned closer to theoccupant 210 in the vehicle 200 as the speed of the vehicle 200increases beyond the local speed limit and to cause the y controllingone or more parameters of the display element to cause therepresentation to be spatially positioned further from the occupant 210in the vehicle 200 as the speed of the vehicle 200 decreases below thelocal speed limit.

For example, icon 232 is shown to be displayed in configuration 232B, sothat the icon 232 is displayed as being positioned a particular distancefrom the vehicle 200 along roadway 292, where the vehicle speed matchesthe local speed limit, shown here as 35 mph. In some embodiments, thedisplay 230 can be controlled to cause the displayed icon 232 to bepositioned 234 within a certain portion of the roadway 292, so that theelement 232 is repositioned in the display 230 to cause thethree-dimensional representation of the graphical icon indicating thevehicle speed to appear to follow various turns, slopes, etc. of theroadway 292 while remaining a particular perceived distance from thevehicle in the environment 290.

As shown in FIG. 2A-B, one or more parameters of the element 232 can beadjusted, via adjusting one or more parameters of the displayed element232, so that the three-dimensional representation of the icon ispositioned in a configuration 232A in the environment which is locatedcloser to vehicle 200 as the vehicle speed increases beyond the localspeed limit. The repositioning 234 can be dynamic, so that the icon isdynamically spatially repositioned in the environment 290 in accordancewith the speed of the vehicle 200. As shown in FIG. 2B, spatialrepositioning can include adjusting a size and position of the element232 so that the representation of the icon appears larger and nearer asvehicle speed is greater than the local speed limit and smaller andfurther away as the vehicle speed is less than the local speed limit.The spatial positioning of the graphical icon in the environmentprovided by the three-dimensional representation of the icon presentedby the display element 232 can provide the occupant 210 with a visualcue with regards to the relative speed of the vehicle 200, therebyaugmenting the occupant's situational awareness, particularly where theoccupant 210 is a driver of the vehicle 200, thereby augmenting vehiclenavigation by the driver occupant 210.

As shown in FIG. 2A-B, the repositioning 234 of the perceived positionof icon in the environment 290 by element 232 can be adjusted to cause aperceived elevation of the element 232 in the environment to be adjustedbased on the relative speed of the vehicle, relative to a local speedlimit. As shown in FIG. 2A, for example, parameters of the displayedicon presented by element 232 can be adjusted to cause the displayedicon to appear at a higher elevation in accordance with the vehiclespeed being greater than a local speed limit, as shown at 232A, and tocause the displayed icon, which can also be referred to herein as arepresentation of the vehicle speed, to appear at a lower elevation inaccordance with the vehicle speed being less than a local speed limit,as shown at 232C. Such adjustment of perceived elevation can occurconcurrently with adjustment of a perceived proximity of the icon fromthe vehicle, as shown in FIG. 2A.

FIG. 3A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which modifies occupant-perceivedinformation presented by a sign located in the external environment,according to some embodiments. The vehicle and transparent surfaceillustrated in FIG. 3A-B can be included in any of the embodiments ofvehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display system generates, on atransparent surface, an augmented reality display of a portion of anexternal environment which can be perceived via the transparent surface,where the augmented reality display comprises display elements whichoverlay one or more portions of the environment which include one ormore objects in the environment and which at least partially conform toone or more boundaries of the one or more objects in the environment.The display elements can overlay content included on the externalsurfaces of the objects and can include replacement content which can begenerated by the augmented reality display system, such that theaugmented reality display element which overlays at least a portion ofthe object, as perceived via the transparent surface, obscures at leasta portion of the content included on the object in the environment withaugmented reality content generated by the augmented reality displaysystem.

As shown in FIG. 3A-B, vehicle 300 in which occupant 310 is locatedincludes a transparent surface 320 and an external sensor 340 whichmonitors 342 one or more portions of environment 390 in which vehicle300 is located, including a roadway 392 along which the vehicle 300 isnavigated. Based on sensor data representations of the monitoredenvironment, which are generated by the sensors 340, an augmentedreality display system included in vehicle 300 can identify one or moreobjects located in the environment 390, including a roadway 392 and roadsigns 350, 370. Based on processing the sensor data, the augmentedreality display system can identify content 352, 372, 374 presented onthe signs. For example, the system can identify sign 350 as a speedlimit sign and further identify that content 352 presented on the sign350 includes a local speed limit of roadway 392, and the system canfurther identify sign 370 as a distance indication sign and furtheridentify that content 372 presented on the sign includes an indicationof a distance to a particular landmark and that content 374 presented onthe sign includes an indication of a distance to another particularlandmark.

In some embodiments, the augmented reality system included in vehicle300 generates an augmented reality display, on surface 320, whichincludes display elements 360, 380 which overlay and conform to one ormore boundaries of one or more particular portions of particular objects350, 370 in the environment 390.

The display elements can be selectively generated, and can includecontent selectively generated by the augmented reality display system,based on monitoring one or more of personal data associated with theoccupant, a driving route along which the vehicle is being navigated,the environment 390, etc. For example, the augmented reality displaysystem can determine, based on monitoring environmental conditions viasensors 340, that environmental conditions, that environmentalconditions are associated with an adjustment to the local speed limit.Where the present environmental conditions are determines to include arainstorm, for example, the augmented reality system can determine,based on comparing the determined environmental condition with adatabase of speed limit adjustments and associated environmentalconditions, that the local speed limit is to be adjusted downwards by 5mph. As a result, the augmented reality display system can generate adisplay element 360 which is perceived by occupant 310 to overlay andconform to the dimensions, boundaries, etc. of the content-includingportion of sign 350 and includes content 362 which includes a localspeed limit which is adjusted from the speed limit 352 actually includedon sign 350 in accordance with the detected environmental condition. Thedisplay element 360 can provide a representation of the sign 350 whichoverlays the portion of the sign 350 which includes content 352, so thatthe content 352 is obscured and content 362 is visible when the sign 350is perceived via the augmented reality display presented on surface 320.

In another example, shown in FIG. 3B, the augmented reality displaysystem can process distance information presented on sign 370 andcompare the information 372, 374 with a driving route along which thevehicle 300 is being navigated. The augmented reality display system cangenerate one or more display elements which selectively highlight one ormore indicated landmarks, directions, etc., selectively obscure one ormore indicated landmarks, directions, etc. based on the driving route.In some embodiments, the system can translate content presented on asign into another language, including a language associated with one ormore of the vehicle, an occupant included in the vehicle, personal dataassociated with an occupant included in the vehicle, etc., and generatea display element which overlays the content and provides a translatedversion of the content on the sign.

As shown in FIG. 3B, based on a determination that content 372 isassociated with the present driving route along which the vehicle 300 isbeing navigated, for example, a destination of the driving route, acheckpoint along the driving route, some combination thereof, etc., theaugmented reality display system included in vehicle 300 can generate adisplay element 380 which has a particular size, shape, and position onsurface 320 to cause the element 380, when perceived by occupant 310 viasurface 320, to overlay and conform to a portion of sign 370 on whichcontent 374 which is unrelated to the driving route is displayed, sothat the element 380 obscures the unrelated content 374 so that theperceived content on the sign 370 is restricted to the relevant content372.

In some embodiments, based on a determination that one or more instancesof content on the 370 is related to a driving route along which thevehicle 300 is being navigated, the augmented reality display system cangenerate a display element which highlights one or more instances ofcontent displayed on a sign in the environment.

Display elements presented on surface 320 as part of the augmentedreality display on surface 320 can be generated based on sensor datagenerated by sensors 340, where the display elements are adjusted basedon the dynamic relative positions of the objects 350, 370 to vehicle300, so that the display elements 360, 380 are perceived to remainconformed to the objects 350, 370 as the vehicle changes positionrelative to the objects.

In some embodiments, the augmented reality display system generateddisplay elements on surface 320 based on personal data associated withoccupant 310. For example, the display system can determine, based onprocessing sensor data representations of sign 370, that content 372indicates a landmark which is a commonly-navigated destinationassociated with personal data of the occupant 310. As a result, theaugmented reality display system can provide an augmented realitydisplay of the environment 390 which is relevant to the occupant's 310interests.

Users can selectively block use of, or access to, personal data. Asystem incorporating some or all of the technologies described hereincan include hardware and/or software that prevents or blocks access tosuch personal data. For example, the system can allow users to “opt in”or “opt out” of participation in the collection of personal data orportions of portions thereof. Also, users can select not to providelocation information, or permit provision of general locationinformation (e.g., a geographic region or zone), but not preciselocation information.

Entities responsible for the collection, analysis, disclosure, transfer,storage, or other use of such personal data should comply withestablished privacy policies and/or practices. Such entities shouldsafeguard and secure access to such personal data and ensure that otherswith access to the personal data also comply. Such entities shouldimplement privacy policies and practices that meet or exceed industry orgovernmental requirements for maintaining the privacy and security ofpersonal data. For example, an entity should collect users' personaldata for legitimate and reasonable uses, and not share or sell the dataoutside of those legitimate uses. Such collection should occur onlyafter receiving the users' informed consent. Furthermore, third partiescan evaluate these entities to certify their adherence to establishedprivacy policies and practices.

FIG. 4A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which includes a display elementwhich provides a representation of a static object in the environmentwhich is presently obscured from direct view through the transparentsurface of the vehicle, according to some embodiments. Therepresentation can include a three-dimensional representation of thestatic object in the environment. The vehicle and transparent surfaceillustrated in FIG. 4A-B can be included in any of the embodiments ofvehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display system generates anaugmented reality display, on a transparent surface, which includesdisplay elements providing representations of objects, in an environmentperceivable via the transparent surface, which are at least partiallyobscured from direct observation via the transparent surface.

In some embodiments, the augmented reality display system included in avehicle 410 navigating along a roadway 400 through environment 490receives information regarding various static objects, including one ormore structures, signs, landmarks, plants, etc., in the environment 490,via a communication link 452 with a remotely located system 450. Theinformation can include location information, dimensions, etc.associated with an object. For example, where environment 490 includesstatic object 430 located adjacent to roadway 400, informationindicating the location and dimensions of object 430 can be received atvehicle 410 via link 452.

In some embodiments, the augmented reality display system included invehicle 410 determines a blind spot 440 associated with sensor datarepresentations of the environment 490 which are generated by sensordevices included in the vehicle 410, based on a presence of one or morevarious objects, including the illustrated vehicle 420, in theenvironment. In some embodiments, based on a determination that adynamic object 420 is at least partially obscuring perception of astatic object 430 via a transparent surface 480 in the vehicle 410, theaugmented reality display system included in the vehicle can generate anaugmented reality display which includes display elements 492-496 whichcomprise representations of the position and boundaries of one or moreportions of one or more obscured elements in the environment 492,including an element 492 which indicates at least a portion of theobscured static object 430. As also shown in FIG. 4B, the displayelements presented via the transparent surface 480 can includerepresentations of portions 494-496 of the roadway which are obscured byobject 420.

In some embodiments, the augmented reality display system selectivelygenerates display elements 492 which comprise representations of one ormore of a position, boundary, etc. of at least a portion of an obscuredstatic object in the environment based on a determination that thestatic object is associated with one or more of a driving route alongwhich the vehicle is being navigated, one or more instances of personaldata associated with one or more of the occupants of the vehicle 410,some combination thereof, etc.

FIG. 5A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which provides a representation ofat least one dynamic object in the environment which is presentlyobscured from direct view through the transparent surface of thevehicle, according to some embodiments. The vehicle and transparentsurface illustrated in FIG. 5A-B can be included in any of theembodiments of vehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display system included in avehicle 510 navigating through an environment 590 which includes aroadway can generate, on a transparent surface 580 included in thevehicle 510, an augmented reality display element 596 which represents aposition and dimension of a dynamic element 530, in the environment 590,which is at least partially obscured 560 from monitoring by vehicle 510by another dynamic element 520 in the environment 590.

In some embodiments, the augmented reality display system receivesinformation associated with the obscured object 530, including sensordata representations of the object 530, based on communication with oneor more of a remote system 550 via a communication link 552,vehicle-to-vehicle (“V2V”) communication with one or more vehicles 520,540 which can directly monitor the object 530 via links 553, etc. Theaugmented reality display system can generate the augmented realitydisplay elements 592-596, including the element 596 which indicates aposition and dimensions of the dynamic element 430 in the environment,based on information received from one or more of the vehicles 520, 540,systems 550, etc.

FIG. 6A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which provides a dynamicallycontrollable representation of a driving route along which a vehicle isbeing navigated, according to some embodiments. The vehicle andtransparent surface illustrated in FIG. 6A-B can be included in any ofthe embodiments of vehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display system included in avehicle generates an augmented reality display, on one or moretransparent surfaces of the vehicle, which includes display elementswhich comprise a representation of a driving route along which thevehicle is being navigated. As referred to herein, a representationcomprised by a display element can include a three-dimensionalrepresentation. The display elements can comprise representations ofvarious navigation actions which can be executed by control elements ofthe vehicle, including braking, accelerating, turning, etc.

As shown in FIG. 6A-B, where vehicle 600 navigates through anenvironment 690 which includes roadways 692-696, a set of one or moreexternal sensors 640 monitors 642 the environment, including theroadways 692-694 and generates sensor data representations of same. Anaugmented reality display system included in vehicle 600 can, based onthe sensor data representations of the environment and a driving route,through the environment, along which the vehicle 600 is being navigate,generates an augmented reality display on a transparent surface 620 ofthe vehicle 600. The augmented reality display includes elements 662,664 which provide visual indications of the route along which thevehicle 600 is being navigated. The display elements can be adjustablycontrolled to cause the elements 662, 664 to be perceived as beinglocated in the environment 690 along the driving route, and parametersof the display elements can be dynamically adjusted as the vehicle 600navigates along the route to maintain a three-dimensional appearance ofthe elements 662, 664 on the route.

As shown, the displayed elements 662, 664 on the surface are arranged onthe roadways 692-696 to illustrate a driving route which proceeds alongroadway 692 to roadway intersection 694, whereupon the route turns tothe left to roadway 696, thereby indicating that the driving routeproceeds along the course illustrated in the display by elements 662,664. In addition, the display elements 662, 664 are displayed to beperceived as being spatially positioned along the route in accordancewith the acceleration, velocity, etc. of the vehicle 600 along theroute. As shown, the elements 662 are displayed as being spatiallypositioned proximate to each other proximate to roadway 692 terminatingat intersection 694, thereby providing a visual indication, to occupant610, that the driving route represented by elements 662 includes thevehicle 600 decelerating at the intersection of roadway 692 with roadwayintersection 694, including providing a visual indication that thevehicle 600, being navigated along the represented driving route, willstop at the threshold of intersection 694. In addition, elements 694 aredisplayed so that the elements are perceived as being spaced furtherapart in the environment 690, relative to elements 662, therebyproviding a visual indication, to occupant 610, that the vehicle 600will accelerate through roadways 694-696 subsequent to decelerating tothe threshold between roadways 692, 694.

In some embodiments, an augmented reality display system dynamicallycontrols the display elements included in an augmented reality displaywhich is presented on one or more transparent surfaces based onmonitoring one or more occupants located proximate to the one or moretransparent surfaces. The system can monitor occupant identity, stresslevel, cognitive load, etc. based on processing sensor data generated byone or more internal sensors 630 included in the vehicle 600 whichmonitor 632 one or more portions of the vehicle 600 interior in whichone or more occupants 610 are located. Monitoring occupant identity caninclude comparing a sensor data representation of an occupant 610 withpersonal data to associate an occupant 610 of the vehicle with aparticular user profile, account, etc. Augmented reality displays can beadjusted based on occupant display preferences which may be included ina user account, profile, etc. associated with a particular occupantidentity.

In some embodiments, the augmented reality display system can adjust thedisplay elements presented in the augmented reality display which can beperceived by a monitored occupant 610 based on an interaction history,indicated in personal data associated with the monitored occupant 610,where the interaction history indicates a historical record ofinteraction between the occupant 610 and one or more of the vehicle 600,the augmented reality display system included in the vehicle, one ormore particular features of the vehicle 600, including autonomousnavigation of the vehicle along a driving route, some combinationthereof, etc. For example, where the augmented reality display systemdetermines, based on identifying occupant 610 and processing personaldata associated with the occupant, that the occupant is associated witha quantity of interactions with features of the vehicle 600 associatedwith navigating the vehicle 600 along a driving route, and a display ofelements 662, 664 on surface 620 which are associated with navigationalong the route, which at least meets a threshold quantity, theaugmented reality display system may refrain from displaying elements662, 664 in an augmented reality display on surface 620, as the occupantmay be determined to be sufficiently experienced with navigation along adriving route in vehicle 600 that the occupant 610 is determined to notrequire visual indications that the vehicle 600 is navigating along adriving route. In another example, where the augmented reality displaysystem determines, based on identifying occupant 610 and processingpersonal data associated with the occupant, that the occupant isassociated with a quantity of interactions with features of the vehicle600 associated with navigating the vehicle 600 along a driving route,and a display of elements 662, 664 on surface 620 which are associatedwith navigation along the route, which does not meet a thresholdquantity, the augmented reality display system may display elements 662,664 in an augmented reality display on surface 620, as the occupant maybe determined to be insufficiently experienced with navigation along adriving route in vehicle 600 that the occupant 610 is determined torequire visual affirmation that the vehicle 600 is navigating along adriving route.

In some embodiments, monitoring a stress level, cognitive load, etc. ofan occupant can include monitoring one or more various features of anoccupant, including eye motion, body posture, body gestures, bodytemperature, breathing rate, eye blink rate, head motion, eye motion,heart rate, some combination thereof, etc. Based on determining that oneor more monitored features of an occupant 610 is associated with adetermined value which at least meets a threshold level associated witha particular stress level, cogitative load level, etc. the augmentedreality display system can determine that a monitored occupant 610 isassociated with the particular stress level, cognitive load level, etc.

The augmented reality display system can dynamically adjust theaugmented reality display, including which display elements arepresented in the display, based on the monitored stress level, cognitiveload level, etc. of one or more of the monitored occupants 610. Forexample, where an occupant 610 is determined to be associated with ahigh cognitive load level, the augmented reality display system canadjust the augmented reality display to remote at least some of thedisplay elements, for example removing elements 662, 664, to cause theoccupant 610 cognitive load to be reduced. In another example, where anoccupant 610 is determined to be associated with a high stress level,the augmented reality display system can adjust the augmented realitydisplay to provide at least some elements, including elements 662, 664,which are associated with control of the vehicle, thereby providingindications to the occupant 610 of the navigation actions being executedduring navigation of the vehicle and assurance that the vehicle is beingnavigated along a driving route to a destination.

FIG. 7A-B illustrate a vehicle which simulates, via one or more of atransparent surface which provides an augmented reality display and anactive suspension system, one or more objects in the environment,according to some embodiments. The vehicle and transparent surfaceillustrated in FIG. 7A-B can be included in any of the embodiments ofvehicles and transparent surfaces included herein.

In some embodiments, an augmented reality display system can display, inan augmented reality display on a transparent surface, a display elementwhich comprises a representation of a simulated environmental object inthe perceived environment. The representation can include athree-dimensional representation. The simulated environmental object canbe represented based on monitoring manual navigation of a vehicle by anoccupant of the vehicle, where the simulated environmental object isgenerated to cause the occupant to adjust the manual driving performanceof the occupant. The augmented reality display system can monitor thedriving performance of an occupant based on processing sensor datagenerated by one or more external sensors, internal sensors, etc.included in the vehicle.

For example, as shown in FIG. 7A-B, an augmented reality system includedin vehicle 700, which is navigating along roadway 790 throughenvironment 721, can determine, based on processing sensor datagenerated by one or more external sensors 740 monitoring 742 theenvironment 721, one or more internal sensors 730 monitoring 732 one ormore portions of the vehicle 700, some combination thereof, etc., thatthe occupant 710 is navigating the vehicle 700 along roadway 790 at aspeed which at least meets an excessive speed threshold value. Inresponse, the augmented reality display system can generate an augmentedreality display, presented on surface 720, which includes one or moredisplay elements 780 which comprise a three-dimensional representationof a simulated environmental object, thereby simulating the presence ofan environmental object, in the environment 721, which is associatedwith reducing vehicle speed. The simulated environmental object 780shown in FIG. 7A-B simulates the presence of a “speed bump” element in aparticular position on the roadway 790. The augmented reality displaysystem can initially generate the display element 780 to cause thesimulated environmental object to be simulated to be positioned at aparticular position on the roadway which is at a distance ahead of thevehicle 700, based on one or more of the present vehicle 700 speed, anestimated reaction time of the occupant 710, etc. The displayed element780 can be dynamically adjusted in the display on surface 720 as thevehicle 700 is navigated along roadway 790, so that the occupant 710monitoring 712 the environment 721 through surface 720 can perceive thesimulated speed bump 780 as a static element on the roadway 790 whichremains at the particular position on the roadway as the vehicle 700approaches the particular position. The augmented reality display systemcan display one or more simulated environmental object display elements780 as the vehicle is navigated in response to manual navigation of thevehicle, occupant driving performance, etc., meeting one or more variousthresholds, and the display of such simulated environmental objectdisplay elements 780 can be precluded in response to the manualnavigation of the vehicle, occupant driving performance, etc., beingdetermined to be below one or more various thresholds.

In some embodiments, the augmented reality display system cancommunicate with one or more control elements of the vehicle 700 toprovide additional simulation of an environmental object in theenvironment 721. As shown, vehicle 700 can include a set of wheels 760which are coupled to the remainder of the vehicle 700 at least partiallyby one or more suspension systems 750. In some embodiments, one or moreof the suspension systems comprises an active suspension system whichcan actuate to simulate one or more of the wheels 760 passing over oneor more environmental objects in the environment 721. In someembodiments, the augmented reality display system in vehicle 700, inaddition to generating an augmented reality display element 780 whichsimulates a static object in the environment 721, can command one ormore active suspension 750 control elements in the vehicle 700 toactuate concurrently with the vehicle 700 passing over the position, inthe environment 721, in which the simulated environmental object issimulated to be located, so that the active suspension system 750simulates the vehicle 700 passing over the simulated environmentalobject. The active suspension systems 750 can be commanded to actuatedifferently based on different simulated environmental objects. Forexample, in the illustrated embodiment, where the augmented realitydisplay system simulates a “speed bump” environmental object at aparticular position in the roadway via display element 780, theaugmented reality display system can command one or more of the activesuspension systems 750 to actuate the wheels 760 in a sequence whichsimulates the vehicle 700 passing over a large obstacle in the roadway790 when the vehicle passes over the location in the roadway 790 inwhich the simulated speed bump is represented to be located by displayelement 780. In another example, where the augmented reality displaysystem simulates a sequential series of small “rumble strip”environmental objects arranged in parallel transversely across aparticular position in the roadway via multiple display elements 780,the augmented reality display system can command one or more of theactive suspension systems 750 to actuate the wheels 760 in a sequencewhich simulates the vehicle 700 passing over a series of small obstaclesin the roadway 790 when the vehicle passes over the location in theroadway 790 in which the simulated rumble strips are represented to belocated by display elements 780.

In some embodiments, the augmented reality display system dynamicallyadjusts one or more augmented reality display elements which arepresented on a transparent surface 720 based on monitoring navigationperformance of the vehicle 700. For example, as shown in FIG. 7B, theaugmented reality display system can generate display elements 781, 783which conform to the lane boundaries 782, 784 of the lane 785 in theroadway 790 along which the vehicle 700 is presently being navigated,and the augmented reality display system can dynamically adjust one ormore parameters of the display elements 781, 783 based on monitoring,via processing sensor data generated by one or more sensors 740, 730,one or more of the vehicle 700 position relative to one or more of thelane boundaries 782, 784, environmental conditions, occupant cognitiveload, etc.

For example, in response to determining that ambient light levels in theenvironment 721 are below a threshold ambient light level, the augmentedreality display system can adjust one or more of the color, brightness,etc. of elements 781, 783 to cause the lane boundaries to be more easilyperceptible by occupant 710 monitoring 712 the environment 721 throughsurface 720. In another example, the augmented reality display systemcan dynamically adjust one or more of color, brightness, pattern,animation sequence, etc. of separate elements 781, 783 based on relativedistance between vehicle 700 and the respective lane boundaries 782, 784associated with the separate elements 781, 783, to that the occupant 710is provided, via the augmented reality display on surface 720, with avisual indication that the vehicle 700 is approaching one or more of thelane boundaries 782, 784.

Such visual indications can augment driver situational awareness and canmitigate a risk of the vehicle 700 drifting out of the lane 785 in whichit is presently navigating.

FIG. 8A-B illustrate a vehicle comprising a transparent surface whichprovides an augmented reality display which includes a representation ofa particular driving zone in the environment, according to someembodiments. The vehicle and transparent surface illustrated in FIG.8A-B can be included in any of the embodiments of vehicles andtransparent surfaces included herein.

In some embodiments, an augmented reality display system installed in avehicle navigating through an environment can identify a particular zonein the environment and can, in response, generate an augmented realitydisplay which comprises an overlay of one or more portions of theenvironment associated with the zone which can be perceived through atransparent surface.

In some embodiments, a particular zone can include one or more of aschool zone, a crossing zone, a zone associated with a presence ofchildren, a zone associated with differently-abled individuals, somecombination thereof, etc. The augmented reality display system candetermine a presence of a particular zone associated with one or moreparticular individuals, structures, events, etc., including determiningone or more boundaries of the particular zone in the environment, basedon one or more of communication with one or more remotely locatedvehicles, devices, systems, services, etc., monitoring of one or moreportions of the environment by one or more sensors included in thevehicle, some combination thereof, etc.

FIG. 8A-B illustrate a vehicle 810 navigating through an environment 800along a roadway 801 in which a region 820 is located. The region 820 caninclude one or more structures 822, individuals 826, zone identificationelements 824A-B, some combination thereof, etc. The augmented realitydisplay system included in vehicle 810 can, based at least in part upona determination that region 820 is associated with a particular zone,generate an augmented reality display 812, presented on a transparentsurface 811 of the vehicle 810, which includes a display element 880which highlights a portion of the roadway 801 which overlaps with thezone 820. In some embodiments, the display element 880 overlaps anentirety of the zone 820. The highlighted portion of the zone indicatedby the display element 880 can conform to one or more boundaries of thezone in the environment, as shown by the highlighted portion of the zoneindicated by display element 880 which terminates, at opposite ends, atboundaries of the zone 820. As a result, the augmented reality displaysystem provides, to one or more vehicle occupants perceiving theenvironment via the transparent surface on which the display isprovided, a visual indication of the particular zone, which can augmentoccupant situational awareness of environmental features associated withthe zone and can augment safe navigation of the vehicle through theenvironment. For example, where the particular zone is a zone associatedwith the presence of children, a visual indication of the zone in anaugmented reality display can augment situational awareness, by a driverof the vehicle, of the possibility that children may be present in theenvironment, thereby augmenting safety to both the driver and anychildren in the zone.

In some embodiments, the augmented reality display system determines alocation and one or more boundaries of a zone based on identifying oneor more zone identification elements 824A-B in the environment. A zoneidentification element 824 can include one or more of a sign, visualicon, symbol, transponder beacon, etc. which indicates one or more of apresence and boundary of the zone 820. Identifying the zoneidentification element can include detecting a presence of the object inthe environment 824, processing content comprised in the element (e.g.,text included on a sign element 824, data comprised in a signalgenerated by a transponder beacon element 824, etc.), and identifyingone or more of the presence of the zone, one or more boundaries of thezone, etc. based on processing the content.

In some embodiments, the augmented reality display system determines alocation and one or more boundaries of a zone based on identifying oneor more features located in the environment which are determined to beassociated with one or more of the presence of the particular zone andone or more boundaries of the zone. Features which can be identified caninclude one or more static elements, dynamic elements, etc. which can bedetected in the environment based on processing sensor datarepresentations, of the environment, which are generated by one or moreexternal sensors included in the vehicle.

In some embodiments, where environment 800 includes structure 822, anaugmented reality display system included in vehicle 810 can identifyzone 820 based on detecting the structure 822 in the environment 800,based on processing sensor data generated by external sensors in thevehicle 810, and determining that the structure 822 is associated withone or more particular zones. For example, the augmented reality displaysystem can determine that the detected structure 822 is a schoolstructure and can determine, based at least in part upon determiningthat structure 822 is a school structure, that the structure 822 isassociated with a school zone 820 in the environment 800 and can, inresponse, generate an augmented reality display which includes a displayelement 880 which overlays a portion of the roadway 801 which overlapsthe school zone 820.

In some embodiments, where environment 800 includes one or moreindividuals 826, an augmented reality display system included in vehicle810 can identify zone 820 based on detecting the one or more individuals826 in the environment 800, based on processing sensor data generated byexternal sensors in the vehicle 810, and determining that the one ormore individuals 826 are associated with one or more particular zones.For example, the augmented reality display system can determine that thedetected individuals 826 are human child individuals and can determine,based at least in part upon determining that individuals 826 are humanchild individuals, that the portion of the environment in which theindividuals 826 are located is associated with a child zone 820 in theenvironment 800 and can, in response, generate an augmented realitydisplay which includes a display element 880 which overlays a portion ofthe roadway 801 which overlaps the child zone 820.

In some embodiments, where environment 800 includes one or more remotelylocated elements 830, which can include one or more remotely locatedsystems, services, vehicles, devices, etc., an augmented reality displaysystem included in vehicle 810 can identify zone 820 based oncommunicating 832 with the remotely located element 830 via one or morecommunication networks and receiving, from the element 830, informationidentifying one or more of the location and one or more boundaries ofthe zone 820 in the environment 800. For example, the augmented realitydisplay system can determine an identity, location and boundaries ofzone 820 based on receiving information from element 830 via aninterface, included in the vehicle 810, which can include one or morecommunication interfaces. In some embodiments, information identifying alocation, boundaries, etc. of the zone at element 830 is generated basedon sensor data generated by one or more vehicles 810, 840, 850navigating through environment 800 and communicating said data toelement 830 via one or more communication links. The element candetermine one or more of an identity, location, boundaries, etc. of thezone 820 based on processing sensor data generated by the variousvehicles navigating through the area, and the determined identity,location, boundaries, etc. of the zone 820 can be communicated to one ormore vehicles navigating through the environment 800.

In some embodiments, the display element 880 is associated with theparticular zone identified in the environment, and can include contentindicating the identity of the zone. For example, element 880 can beassociated with one or more colors, animation sequences, instances oftext content, instance of audio content, instances of video content,some combination thereof, etc. which indicates the identity of theparticular zone indicated by the element 880 (e.g., school zone).

FIG. 9 illustrates a transparent surface which provides an augmentedreality display of one or more portions of an environment, according tosome embodiments. FIG. 10 illustrates a vehicle comprising a transparentsurface which provides an augmented reality display which enables userselection of a portion of the environment perceived via the transparentsurface and which enables video communication with a remotely-locateduser via the transparent surface, according to some embodiments. Thevehicle and transparent surface illustrated in FIG. 9-10 can be includedin any of the embodiments of vehicles and transparent surfaces includedherein.

In some embodiments, an augmented reality display system, in response toassociating an element in the external environment with a particularuser profile, account, etc. associated with a particular user, generatesan augmented reality display element, presented on a transparentsurface, which identifies the environmental object and includes anavatar icon associated with the particular user profile, account, etc.,thereby providing a visual indicator of the user to an occupantperceiving the environment via the transparent surface. The augmentedreality display system can selectively generate the avatar displayelement based on a determination that the particular user is associatedwith an occupant positioned proximate to the transparent surface throughwhich the environmental object, which can include one or more staticelements, dynamic elements, etc., can be perceived.

For example, where the augmented reality display system included in avehicle (the “ego-vehicle”) determines, based on processing externalsensor data, that a vehicle 940 navigating proximate to the ego-vehicleis associated with a user profile of a particular user associated with auser profile of an occupant 1010 of the vehicle 1000. In response, theaugmented reality display system can generate, on transparent surface900 of vehicle 1000 through which occupant 1010 can perceive the vehicle940, a display element 942 which identifies the vehicle 940 as beingassociated with the particular user. The element 942 can include anavatar icon which is associated with the particular user, therebyproviding a recognizable visual indication of an association between theelement 940 and the particular user.

In some embodiments, the augmented reality display system generates, onthe transparent surface 900, a display element 930 which simulates oneor more of the location, dimensions, etc. of a future object in theenvironment which is under construction, proposed for construction, etc.For example, where a future element includes a structure which ispresently under construction, the augmented reality display systeminclude in vehicle 110 can generate an augmented reality display 910, onsurface 900, which includes a display element 930 which includes arepresentation of the outline and location, in the environment, of thefuture structure.

In some embodiments, display of elements 942, 930 can be selectivelyimplemented based on one or more instances of personal data associatedwith occupant 1010 which indicate whether to generate display elements930 indicating future environmental objects, user identificationelements 942, some combination thereof, etc.

In some embodiments, an augmented reality display system generates adisplay element, on a transparent surface, which identifies(“highlights”) an element located in the environment, so that anoccupant perceiving the object in the environment via the transparentsurface perceives that the element is identified by a display elementpresented on the transparent surface.

In some embodiments, the augmented reality display system generates adisplay element 960 which identifies an environmental object 950 basedon determining, as a result of processing a sensor data representationof an occupant, that the occupant is selecting the particularenvironmental object in the environment. As shown in FIG. 10 , thevehicle 1000 can include one or more internal sensors 1030 which monitor1032 a portion of an interior of the vehicle 1000 in which an occupant1010 is located. The augmented reality display system can monitor one ormore various body gestures, eye movements, etc. of the occupant 1010based on processing internal sensor data representations of the occupant1010 which are generated by the sensors 1030. The system can process theinternal sensor data representations, in addition to external sensordata representations generated by one or more external sensors includedin the vehicle 1000 and can, based on the processing, determining thatthe occupant 1010 is selecting a particular environmental object 950located in the environment 910. Such a determination can include adetermination that the occupant 1010 is pointing a limb 1012 in adirection towards the position of the element 950 in the environment, asperceived by the occupant 1010 via the transparent surface 900, adetermination that one or more portions of the occupant 1010, includingone or more eyes of the occupant 1010, are pointed towards the element950 in the environment, some combination thereof, etc.

In some embodiments, an augmented reality display system can generatevarious display elements to enable occupant interaction with one or moreportions of the environment. For example, where an environmental object950 is identified based on a determination that occupant 1010 isselecting the element 950, the augmented reality display system cangenerate one or more instances of content, including one or moredrawings, symbols, instances of text content, etc. which are included inan augmented reality display element indicating the environmental object950. Such content can be incorporated into a display elementrepresenting, identifying, etc. the element 950, based on occupant 1010interaction with one or more user interfaces included in the vehicle1010.

In some embodiments, an augmented reality display system can generatevarious display elements, which are presented on a transparent surfaceof the vehicle, as part of implementing an entertainment program, game,etc. For example, based on receiving a command, as a result of occupant1010 interaction with a user interface included in the vehicle 1000, toplay an instance of video content, the augmented reality display systemcan generate display elements which present the video content on atransparent surface 900 which can be perceived by the occupant 1010. Inanother example, the augmented reality display system can generatevarious display elements as part of an interactive game, based at leastin part upon occupant actions, gestures, interactions with one or moreuser interfaces, etc.

Based on generating a display element which identifies a particularobject in the environment, the augmented reality display system canadjust the display element to follow the object in the environment,present information associated with the identified element, receiveoccupant commands associated with the identified element, etc.

In some embodiments, an augmented reality display system included in avehicle enables visual communication between an occupant of the vehicleand a remotely located user, including an occupant of a separatevehicle, via generating an augmented reality display, on a proximatetransparent surface of the vehicle, which presents a visual display ofthe remotely located user, based on sensor data generated by a sensordevice monitoring the remotely located user and communicated to thevehicle via one or more communication networks. In some embodiments,where the remotely located user is an occupant of a separate vehiclewhich also includes a transparent surface and an augmented realitydisplay system, the systems of the separate vehicles can each generatean augmented reality display, on a transparent surface of the respectivevehicle, which presents a visual representation, which can include avideo stream, of the user occupying the other vehicle.

As shown in FIG. 10 , for example, an augmented reality display systemcan generate, on surface 900, an augmented reality display whichcomprises a visual representation of a remotely located user 1060, sothat occupant 1010 of vehicle 1000 can visually perceive 1014 user 1060via the surface 900.

FIG. 11 illustrates an example computer system 1100 that may beconfigured to include or execute any or all of the embodiments describedabove. In different embodiments, computer system 1100 may be any ofvarious types of devices, including, but not limited to, a personalcomputer system, desktop computer, laptop, notebook, tablet, slate, pad,or netbook computer, cell phone, smartphone, PDA, portable media device,mainframe computer system, handheld computer, workstation, networkcomputer, a camera or video camera, a set top box, a mobile device, aconsumer device, video game console, handheld video game device,application server, storage device, a television, a video recordingdevice, a peripheral device such as a switch, modem, router, or ingeneral any type of computing or electronic device.

Various embodiments of an augmented reality display system as describedherein, may be executed in one or more computer systems 1100, which mayinteract with various other devices. Note that any component, action, orfunctionality described above with respect to FIGS. 1 through 11 may beimplemented on one or more computers configured as computer system 1100of FIG. 11 , according to various embodiments. In the illustratedembodiment, computer system 1100 includes one or more processors 1110coupled to a system memory 1120 via an input/output (I/O) interface1130. Computer system 1100 further includes a network interface 1140coupled to I/O interface 1130, and one or more input/output devices,which can include one or more user interface (also referred to as “inputinterface”) devices. In some cases, it is contemplated that embodimentsmay be implemented using a single instance of computer system 1100,while in other embodiments multiple such systems, or multiple nodesmaking up computer system 1100, may be configured to host differentportions or instances of embodiments. For example, in one embodimentsome elements may be implemented via one or more nodes of computersystem 1100 that are distinct from those nodes implementing otherelements.

In various embodiments, computer system 1100 may be a uniprocessorsystem including one processor 1110, or a multiprocessor systemincluding several processors 1110 (e.g., two, four, eight, or anothersuitable number). Processors 1110 may be any suitable processor capableof executing instructions. For example, in various embodimentsprocessors 1110 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 1110 may commonly,but not necessarily, implement the same ISA.

System memory 1120 may be configured to store program instructions,data, etc. accessible by processor 1110. In various embodiments, systemmemory 1120 may be implemented using any suitable memory technology,such as static random access memory (SRAM), synchronous dynamic RAM(SDRAM), nonvolatile/Flash-type memory, or any other type of memory. Inthe illustrated embodiment, program instructions included in memory 1120may be configured to implement some or all of an ANS, incorporating anyof the functionality described above. Additionally, existing automotivecomponent control data of memory 1120 may include any of the informationor data structures described above. In some embodiments, programinstructions and/or data may be received, sent or stored upon differenttypes of computer-accessible media or on similar media separate fromsystem memory 1120 or computer system 1100. While computer system 1100is described as implementing the functionality of functional blocks ofprevious Figures, any of the functionality described herein may beimplemented via such a computer system.

In one embodiment, I/O interface 1130 may be configured to coordinateI/O traffic between processor 1110, system memory 1120, and anyperipheral devices in the device, including network interface 1140 orother peripheral interfaces, such as input/output devices 1150. In someembodiments, I/O interface 1130 may perform any necessary protocol,timing or other data transformations to convert data signals from onecomponent (e.g., system memory 1120) into a format suitable for use byanother component (e.g., processor 1110). In some embodiments, I/Ointerface 1130 may include support for devices attached through varioustypes of peripheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard, for example. In some embodiments, the function of I/Ointerface 1130 may be split into two or more separate components, suchas a north bridge and a south bridge, for example. Also, in someembodiments some or all of the functionality of I/O interface 1130, suchas an interface to system memory 1120, may be incorporated directly intoprocessor 1110.

Network interface 1140 may be configured to allow data to be exchangedbetween computer system 1100 and other devices attached to a network1185 (e.g., carrier or agent devices) or between nodes of computersystem 1100. Network 1185 may in various embodiments include one or morenetworks including but not limited to Local Area Networks (LANs) (e.g.,an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., theInternet), wireless data networks, some other electronic data network,or some combination thereof. In various embodiments, network interface1140 may support communication via wired or wireless general datanetworks, such as any suitable type of Ethernet network, for example;via telecommunications/telephony networks such as analog voice networksor digital fiber communications networks; via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices may, in some embodiments, include one or moredisplay terminals, keyboards, keypads, touchpads, scanning devices,voice or optical recognition devices, or any other devices suitable forentering or accessing data by one or more computer systems 1100.Multiple input/output devices may be present in computer system 1100 ormay be distributed on various nodes of computer system 1100. In someembodiments, similar input/output devices may be separate from computersystem 1100 and may interact with one or more nodes of computer system1100 through a wired or wireless connection, such as over networkinterface 1140.

Memory 1120 may include program instructions, which may beprocessor-executable to implement any element or action described above.In one embodiment, the program instructions may implement the methodsdescribed above. In other embodiments, different elements and data maybe included. Note that data may include any data or informationdescribed above.

Those skilled in the art will appreciate that computer system 1100 ismerely illustrative and is not intended to limit the scope ofembodiments. In particular, the computer system and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, Internet appliances,PDAs, wireless phones, pagers, etc. Computer system 1100 may also beconnected to other devices that are not illustrated, or instead mayoperate as a stand-alone system. In addition, the functionality providedby the illustrated components may in some embodiments be combined infewer components or distributed in additional components. Similarly, insome embodiments, the functionality of some of the illustratedcomponents may not be provided and/or other additional functionality maybe available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 1100 may be transmitted to computer system1100 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network and/or a wireless link. Various embodiments mayfurther include receiving, sending or storing instructions and/or dataimplemented in accordance with the foregoing description upon acomputer-accessible medium. Generally speaking, a computer-accessiblemedium may include a non-transitory, computer-readable storage medium ormemory medium such as magnetic or optical media, e.g., disk orDVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR,RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessiblemedium may include transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is:
 1. An apparatus, comprising: an augmented realitydisplay system of a vehicle, wherein the augmented reality displaysystem comprises one or more processors and a memory, wherein theaugmented reality display system is configured to: monitor an objectlocated in an environment in which the vehicle is located and which isperceptible through a transparent surface of the vehicle, wherein theobject comprises instances of content; based on monitoring the object,determine whether respective instances of content are relevant contentor unrelated content with respect to a driving route for the vehicle;and generate an augmented reality display, on the transparent surface,which comprises a display element which conforms to at least oneboundary of at least a portion of the object and at least partiallyobscures one or more instances of the unrelated content on the object,such that one or more instances of the relevant content on the objectare perceptible independently of the one or more instances of theunrelated content via the transparent surface.
 2. The apparatus of claim1, wherein the augmented reality di splay system is configured to:include, in the display element, an instance of generated content, basedon monitoring the object, such that the instance of generated contentreplaces the one or more instances of the relevant content as beingperceptible, via the transparent surface, on the object.
 3. Theapparatus of claim 2, wherein: the instance of generated contentcomprises at least one instance of content, of the one or more instancesof content, which is translated to a different language.
 4. Theapparatus of claim 1, wherein: the display element conforms to allboundaries, of at least a portion of the object, which are perceptiblevia the transparent surface.
 5. The apparatus of claim 1, furthercomprising: one or more sensors configured to generate sensor datarepresentations of the environment, wherein the augmented realitydisplay system is further configured to: monitor the object based atleast on the sensor data representations.
 6. The apparatus of claim 5,wherein the augmented reality display system is further configured to:monitor environmental conditions of the environment based at least onthe sensor data representations of the environment; and generate thedisplay element based at least on the environmental conditions.
 7. Theapparatus of claim 6, wherein the augmented reality display system isfurther configured to: determine that the one or more instances ofcontent on the object include a local speed limit; based on adetermination that the environmental conditions includes rain, include,in the display element, an instance of generated content that is basedat least on an adjusted value of the local speed limit.
 8. The apparatusof claim 1, wherein the augmented reality display system is furtherconfigured to: generate the display element based at least on personaldata associated with an occupant of the vehicle.
 9. One or morenon-transitory, computer-readable storage media storing instructionsthat, when executed on or across one or more processors, cause the oneor more processors to: monitor an object located in an environment inwhich a vehicle is located and which is perceptible through atransparent surface of the vehicle, wherein the object comprisesinstances of content; determine whether respective instances of contentare relevant content or unrelated content with respect to a drivingroute for the vehicle; and generate an augmented reality display, on thetransparent surface, which comprises a display element which conforms toat least one boundary of at least a portion of the perceived object andat least partially obscures one or more instances of the unrelatedcontent on the object, such that one or more instances of the relevantcontent on the object are perceptible independently of the one or moreinstances of the unrelated content via the transparent surface.
 10. Theone or more non-transitory, computer-readable storage media of claim 9,further comprising instructions that, when executed on or across the oneor more processors, cause the one or more processors to: include, in thedisplay element, an instance of generated content, based on monitoringthe object, such that the instance of generated content replaces the oneor more instances of the relevant content as being perceptible, via thetransparent surface, on the object.
 11. The one or more non-transitory,computer-readable storage media of claim 10, wherein: the instance ofgenerated content comprises at least one instance of content, of the oneor more instances of content, which is translated to a differentlanguage.
 12. The one or more non-transitory, computer-readable storagemedia of claim 9, wherein: the display element conforms to allboundaries, of at least a portion of the object, which are perceptiblevia the transparent surface.
 13. The one or more non-transitory,computer-readable storage media of claim 9, further comprisinginstructions that, when executed on or across the one or moreprocessors, cause the one or more processors to: monitor environmentalconditions of the environment based at least on sensor datarepresentations of the environment generated by one or more sensors ofthe vehicle; and generate the display element based at least on theenvironmental conditions.
 14. The one or more non-transitory,computer-readable storage media of claim 13, further comprisinginstructions that, when executed on or across the one or moreprocessors, cause the one or more processors to: determine that the oneor more instances of content on the object include a local speed limit;based on a determination that the environmental conditions includesrain, include, in the display element, an instance of generated contentthat is based at least on an adjusted value of the local speed limit.15. The one or more non-transitory, computer-readable storage media ofclaim 9, further comprising instructions that, when executed on oracross the one or more processors, cause the one or more processors to:generate the display element based at least on personal data associatedwith an occupant of the vehicle.
 16. A method, comprising: monitoring,by an augmented reality display system of a vehicle, an object locatedin an environment in which the vehicle is located and which isperceptible through a transparent surface of the vehicle, wherein theobject comprises instances of content; determining whether respectiveinstances of content are relevant content or unrelated content withrespect to a driving route for the vehicle; and generating, by theaugmented reality display system, an augmented reality display, on thetransparent surface, which comprises a display element which conforms toat least one boundary of at least a portion of the object and at leastpartially obscures one or more instances of the unrelated content on theobject, such that one or more instances of the relevant content on theobject are perceptible independently of the one or more instances of theunrelated content via the transparent surface.
 17. The method of claim16, further comprising: including, in the display element, an instanceof generated content, based on monitoring the object, such that theinstance of generated content replaces the one or more instances of therelevant content as being perceptible, via the transparent surface, onthe object.
 18. The method of claim 17, wherein: the instance ofgenerated content comprises at least one instance of content, of the oneor more instances of content, which is translated to a differentlanguage.
 19. The method of claim 16, wherein: the display elementconforms to all boundaries, of at least a portion of the object, whichare perceptible via the transparent surface.
 20. The method of claim 16,further comprising: monitoring environmental conditions of theenvironment based at least on sensor data representations of theenvironment generated by one or more sensors of the vehicle; generatingthe display element based at least on the environmental conditions;determining that the one or more instances of content on the objectinclude a local speed limit; and based on a determination that theenvironmental conditions includes rain, including, in the displayelement, an instance of generated content that is based at least on anadjusted value of the local speed limit.
 21. The method of claim 16,further comprising: generating the display element based at least onpersonal data associated with an occupant of the vehicle.